Increasingly, multimedia content, such as digital video clips and audio sound bits, is transported over computing device networks, such as the Internet. There are many mechanisms for transporting multimedia content through a computing device network.
Traditionally, multimedia content is transported using a client-server mechanism, in which one or a set of multiple server computing devices are dedicated to hosting and delivering the multimedia content and another one or a set of multiple client devices are dedicated to receiving the multimedia content. In this case, efficiency of the content transportation can be improved by strategically placing intermediate server computing devices, acting as proxies for the server computing devices, caching and delivering the multimedia content to those client devices that are close to them in the network.
Recently, it has become popular to use peer-assisted mechanisms. In this case, the client devices that are receiving the multimedia content themselves replace the intermediate server computing devices in the client-server case as proxies, cache and re-deliver the multimedia content to other client devices. There are still many other content transport mechanisms and efficiency improvement schemes such as single or multiple tree-based application layer multicast, peer-to-peer file downloading, peer-to-peer streaming, and content distribution network (CDN). Collectively, they are referred to as “peer-to-peer” networks (hereinafter referred to as P2P networks).
In addition to multimedia content being playback by the client device only after the entire content file is received, delivery and playback are often done concurrently, also known as “streaming,” regardless of which transportation mechanism or efficiency improvement scheme is used.
For traditional file transfers through a computing device network, a simple quantifiable measurement, such as total delay of the transportation of the file, can be used as an assessment objective. For multimedia content streaming, however, the assessment objective is the quality of the playback experience (hereinafter referred to as QoE), a more elusive and subjective measurement. The only accepted method for measuring QoE is through subjective scoring by representative human audience of the multimedia content being played.
However, a QoE assessment process comprises of multiple subjective scorings is labor-intensive and time-consuming. Ideally, each subjective scoring should be repeated for different settings (network topologies, protocols, algorithms, transport mechanisms, efficiency improvement schemes, and number of audiences, etc), and different types of contents. Furthermore, results of the QoE assessment process should be repeatable to be convincing. In reality, large-scale QoE assessment experiments are very difficult to implement. Some testers use PlanetLab (http://www.planet-lab.org), a collaboration of vast number of computers connected to the Internet around the world, serving as testbed for overlay networks. However, two experiments performed using PlanetLab can hardly be implemented with identical settings such as identical set of computers, much less repeatable results. Another way is to use certain unique content to attract a large audience to participate in an experiment. For example, tests were done during the 2008 Olympics broadcasting. Although such experiments can be reasonably large-scale, they cannot be easily repeated.
Video quality assessments for traditional television contents and related applications have received much attention in the past. There are a number of publications and standards available. For example, International Telecommunication Union, “Recommendation ITU-R BT.500-11—Methodology for the subjective assessment of the quality of television pictures.” (2002) (Contents of which are hereby incorporated by reference) and International Telecommunication Union, “ITU-T Recommendation P.910 Subjective video quality assessment methods for multimedia applications.” (1999) (Contents of which are hereby incorporated by reference). Although some concrete methods designed for subjective assessment of video quality in multimedia applications were discussed in these publications, they do not provide any technical specifications for experimenting on personal computers in computing device networks.
More recently, there are studies on subjective assessments of 3 L-video, namely low-bit rate, low-frame rate, and low-resolution video, which constitutes most of the video content streamed on the Internet. In these assessment studies, each reference sequence (SRC) of content was processed through a number of error conditions, or so called Hypothetical Reference Circuits (HRC), to generate processed video sequences (PVS). The HRC considered so far is suitable for streaming content using client-server mechanism where there is a single link between a server computing device and a client device, and packet loss rate is considered to be the only type of network transmission errors. But such HRC method has limitation in applicability in P2P networks because the HRC method considers packet loss rate as the only type of network transmission error.
When transporting content in P2P networks, the minimum processing unit of the content being transported is no longer packets but chunks. A network packet is a data segment of the content being transported and typically ranges from a few hundreds to tens of thousands of bits in size. A chunk, on the other hand, composes of several consecutive audio and video frames of the content and its length can be from hundreds of milliseconds to one second in uninterrupted playback time. Although the size of a chunk depends on the actual implementation of the particular transport mechanism, the network, and the content streaming application, in any case a chuck is much larger than a packet.
In P2P content streaming applications, almost all the important building blocks and components are designed to handle chunks. For example, chunk selection algorithm, peer selection algorithm that is based on chunk bit-map information exchanged among neighbor peers, and local chunk buffer management. Therefore, in order to assess the QoE of the streamed content transported through P2P networks, innovative methods for generating and extracting chunk-level impairments are needed.
U.S. Patent Application Publication No. 2006/0120463, U.S. Patent Application Publication No. US 2009/0180545, and U.S. Pat. No. 7,266,147 disclose the designs and implementations of Hypothetical Reference Decoders. These disclosures focus on how to evaluate different encoding and decoding methods for the video streaming application without reference to content transportation through the network. On the other hand, the presently disclosed invention is a system and a method for conducting test or experiments of assessing QoE of content playback transported using P2P. It primarily focuses on the chunk-level impairments generated by the network transport components, and neither the media encoder nor the decoder.
In summary, some of the prior arts disclose methods of assessing network transport effects but they were not specific to the transport of streamed multimedia content. Other prior arts disclose methods of assessing network transport effects with specificity on streamed media content, but the methods disclosed focus on packet-level network transport effects and transported using non-P2P network. Still other prior arts disclose methods of transporting media content itself and not of assessing the network transport effects.